Method of processing signals and apparatus for signal processing

ABSTRACT

An image processing apparatus operates to process input signal samples representative of at least part of a color video image to produce legal color signal samples representative of a legal color version of the image. The apparatus comprises an over sampling processor which operates to generate an over sampled version of the input signal samples by generating at least one extra signal sample for each base input signal sample, an adjustment factor generator, which operates to generate a plurality of adjustment factors which when combined with the input signal samples have an effect of converting illegal color pixels of the color image into legal color pixels, a color legalizer coupled to the adjustment factor generator, which operates to combine the adjustment factors with the input signal samples to produce the legalized color signal samples, a decimating processor coupled to the color legalizer which operates to decimate the legalized color signal samples to produce legalized signal samples having a sampling rate corresponding to that of the base input signal samples, a further adjustment factor generator, and a further color legalizer each of which is coupled to the output of the decimating processor, the further adjustment factor generator operating to generate further adjustment factors, and the further color legalizer operating to combine the further adjustment factors with the decimated legalized color signal samples. The image processing apparatus may operate with input signal samples in the form having red, green or blue components, or in a form having luminance and chrominance components, or indeed in other forms.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to methods of processing signal samplesrepresentative of a color video image to produce a legalised colorversion of the image. Furthermore, the present invention relates toapparatuses for processing signal samples representative of a colorvideo image to produce a legalised color version of the image.

2. Description of the Prior Art

It is well known that the colors of the rainbow, which correspond tolight with a range of wavelengths which is visible to the human eye, canbe represented from combinations of the colors red, green and blue. Forthis reason color television and video images are generated byseparating the red, green and blue components of the images and samplingthese components at spatially separated sampling points within theimage. For example, color television cameras are provided with adichronic element which separates the colors of an image formed within afield of view of the camera into red, green and blue components. Each ofthe red, green and blue components of the image is sampled in twodimensions in accordance with a row-by-column de-composition of theimage. Each row is sampled at regularly displaced sampling points toproduce a number of samples representing the row which produces therow-by-column de-composition of the image. These sampling points areknown to those skilled in the art as pixels. Each of the samplesrepresents one of the red, green and blue components of one of thepixels which make up the image.

The color image may be re-generated from the signal samples using acolor visual display unit, by separating the signal samples representingthe red, green and blue components of the pixels and feeding eachrespectively to one of three image generators. Each of the imagegenerators operates to reconstruct, row-by-column, a version of theimage for one of the three colors of red, green or blue which aresuper-imposed on a color screen. By producing the red, green and bluecomponents of each pixel at positions on the screen corresponding to thepositions of the pixels from which the color image was sampled, thecolor image is re-generated. Since each pixel is comprised of red, greenand blue components, the relative intensity of these components producesa mixture of red, green and blue light which represents the color at thecorresponding point of the image. The mixture of the red, green and bluecomponents can therefore reproduce any of the colors of the originalcolor image, which can be any of the colors of the rainbow. A combinedeffect of the three image generators is therefore to reproduce a versionof the color image which is representative of the color image formedwithin the field of view of the television camera.

Representing a color image as red, green and blue signal samplesprovides a facility for transmitting, recording and reproducing thecolor image in some way. However, in order to reduce an amount ofinformation which must be transmitted in order to convey the colorimage, known television transmission techniques and video imagerecording techniques convert the red, green and blue signals into colordifference signals, which are generally comprised of a luminance and afirst and a second chrominance signal. The luminance signal is, forexample, formed by combining the red, green and blue signal componentsof a pixel into a single component representative of the relativestrength of the light in the image at the pixel location. The first ofthe chrominance signals is generated by forming a difference between theluminance signal and the red signal, and the second chrominance signalis formed from the difference between the luminance signal and the bluecolor signal.

The color difference signal format is one example of a signal formatwhich forms a signal space in which the pixels of a color video imagecan be represented, but which does not directly correspond with the red,green and blue components from which the color video image wasgenerated. As a result, not all values of the color difference signalcomponents representing a pixel within the color difference spacecorrespond to pixels within the signal space formed from the red, greenand blue components of the color image. For example, if the luminancecomponent is at its minimum value of zero, then any non-zero value ofthe two chrominance signal components will result in a signal valuewhich does not fall within the red, green and blue color referencespace. Similarly, if the luminance signal is at a maximum value whichcorresponds to white light, then any non-zero values of the twochrominance signals will also not fall within the red, green and bluereference space.

Any color which does not fall within the red, green and blue referencespace is an illegal color. For the example of color difference signals,any combination of the three components of the color difference signalswhich results in a value which does not fall within the red, green andblue color reference space will be an illegal value. Such illegal colorvalues can be produced when the color images are transmitted orprocessed as, for example, color difference signals. For example, videosignals are often processed in this format to introduce video effectssuch as color wash effects. As a result, values of the three colorreference space components can be produced which are illegal valueswithin the red, green and blue reference space. If these illegal colorvalues are displayed within a color image, colors can result which donot match with the legal parts of the image. The color visual displayunit reproducing the image may hard limit the color value to a maximumvalue of the component which can be displayed, and the illegal pixelsmay be reproduced or processed in an unpredictable way.

In an article entitled “Limiting of YUV Digital Video Signals” by V GDevereux from the Research Department, Engineering Division, of theBritish Broadcast Corporation dated December 1987, a method ofconverting illegal color pixels in a form of YUV color differencesignals into legal color pixels with respect to the red, green and blue(RGB) color reference space is disclosed. This method changes thecomponents of the pixels in the YUV color difference space with respectto each other in order to convert the pixel in the corresponding red,green and blue color reference space into a legal pixel.

Having regard to the above discussion, it will be appreciated that thereis a general requirement to provide a method of processing color videoimages in order to convert reliably illegal color pixels of the imagesinto legal color pixels.

SUMMARY OF THE INVENTION

According to the present invention, there is provided a method ofprocessing input signal samples representative of at least part of acolor video image to produce legalised signal samples representative ofa legal color version of the image, the method comprising the steps ofgenerating an over sampled version of the input signal samples bygenerating at least one extra signal sample for each base input signalsample, generating adjustment factors from the input signal sampleswhich when combined with the input signal samples have an effect ofconverting illegal color pixels of the color video image into legalcolor pixels, combining the adjustment factors with the input signalsamples to produce the legalised color signal samples, decimating thelegalised color signal samples to produce legalised signal sampleshaving a sampling rate corresponding to that of the base input signalsamples, generating further adjustment factors in dependence upon thedecimated legalised color signal samples, and combining the furtheradjustment factors with the decimated legalised color signal samples.

It has been discovered that illegal colors can be produced as a resultof distortion caused by aliasing errors. The distortion is produced byhigh frequency components of the video image in the analogue domainwhich are outside a maximum frequency which can be represented inaccordance with a sampling rate of the input signal samples. Thisproduces distortion in the video image as a result of aliasing errors.

To provide an improvement by reducing the effect of this distortion anover-sampled version of the input signal samples is generated so thatthese out-of-band components in the video image appear within thein-band components of the over-sampled version of the input signalsamples. This provides a further advantage in representing the analoguevideo signal more accurately because the sampling points of the videoimage at the lower sampling rate can fall at positions which do notcorrespond to a maximum of the video signal. The video image is thenlegalised in this over sampled form, by generating and applyingadjustment factors to produce legalised color signal samples. However,distortion of the color video image can occur when the over sampledversion of the legalised color signal samples is filtered and decimated,since filtering and decimating the legalised color signal samplesinvolves representing the influence of a plurality of signal samples toproduce a composite decimated signal sample. By performing a furtherlegalising process, resulting from generating further adjustment factorsand combining these with the decimated version of the input signalsamples, any distortion caused by decimating the over sampled version ofthe legalised color signal samples is substantially reduced. The termdecimating refers to a process in which an over sampled signal isreduced to a version with signal samples having a sampling ratecorresponding to that of the original input signal samples. This mayinvolve filtering and then dropping the extra signal samples associatedwith the over sampled version, and forming the decimated version fromthe samples at the same positions as that of the original signalsamples. Decimating in this sense can cause legal color pixels to becomeillegal, as this can involve changing some signal samples.

An effect of applying the further adjustment factors, can be tointroduce aliasing errors in the decimated legalised color signalsamples as a result of an effective expansion of the bandwidth of thevideo signal, which can not be represented with the sampling rate of thedecimated legalised color signal samples. This can cause legal colorpixels or legalised color pixels to become illegal. For this reason, themethod may include the step of softening the further adjustment factorsbefore combining the softened further adjustment factors with thedecimated legalised signal samples. The term soften or softening refersto a process in which the adjustment factors are adapted, changed orprocessed in some way to reduce distortion which the adjustment factorsun-softened can produce in the legalised color signal samples. This maybe performed by filtering the further adjustment factors.

As explained above, although the input signal samples which arerepresentative of the color video image may have values with respect toa signal space which is different from the red, green and blue signalspace, an example embodiment of the invention finds particularapplication where the input signal samples are color difference signalsamples having luminance and two color difference signal components. Assuch, in the case where the adjustment factors are calculated andapplied with reference to the red, green and blue color reference space,the step of combining the adjustment factors with the input signalsamples comprises the steps of converting the input color differencesignal samples into a color reference signal samples having values withrespect to three orthogonal color reference axes of red, green and bluelight, combining the color reference signal samples with the adjustmentfactors and converting the combined color reference signal samples intocolor difference signal samples.

Although the adjustment factors may be digital values which are added tothe input signal samples in order to generate the legalised color signalsamples, in a preferred embodiment, the adjustment factors are scalingfactors and the step of combining the adjustment factors with the inputsignal samples comprises the step of multiplying the adjustment factorswith the input signal samples.

Accordingly to an aspect of the present invention, there is provided animage processing apparatus according to patent claim 6. Further featuresand aspects of the image processing apparatus are provided in theappended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will now be described by way ofexample only with reference to the accompanying drawings, in which:

FIG. 1 is a three-dimensional representation of colors within the red,green and blue color reference space;

FIG. 2 is a three-dimensional representation of the three-dimensionalred, green and blue reference space shown in FIG. 1 within a colordifference signal space;

FIG. 3 is a schematic block diagram of a color television and videoprocessing system;

FIG. 4 is a more detailed schematic block diagram of the imageprocessing apparatus shown in FIG. 3 according to a first exampleembodiment of the present invention;

FIG. 5 is a diagram providing a conceptual representation of theassociation of luminance and chrominance samples in the CCIR-601 4:2:2format;

FIG. 6 is a schematic block diagram of a color format processor whichappears in the image processing apparatus shown in FIG. 4;

FIG. 7 is a schematic block diagram of a more detailed representation ofa rate converter which appears in FIG. 6;

FIG. 8 is a schematic block diagram of a further color legalisingprocessor which is show in FIG. 4;

FIG. 9 is a graphical representation illustrating an effect of thefurther adjustment factors on the video signal;

FIG. 10 is an alternative example of the further color legalisingprocessor shown in FIG. 4;

FIG. 11 is a schematic block diagram of an image processing apparatusaccording to a second embodiment of the present invention;

FIG. 12 is a schematic block diagram of an over-sampling processor whichappears in FIG. 11;

FIG. 13 is a schematic block diagram of an up-sample generator whichappears in FIG. 12;

FIG. 14 is a graphical representation of signal samples produced by theup-sample generator shown in FIG. 13;

FIG. 15 is a schematic block diagram of a color reference converter;

FIG. 16 is graphical representation of the components of color referencesignal samples on corresponding axes;

FIG. 17 is a schematic block diagram of an adjustment factor processorwhich appears in FIG. 11;

FIG. 18 is a schematic block diagram of an adjustment factor softenerwhich appears in FIG. 11;

FIG. 19 is a schematic block diagram of a de-multiplexer which appearsin FIG. 18;

FIG. 20 is graphical representation of signal samples produced by thede-multiplexer shown in FIG. 19;

FIG. 21 is a schematic block diagram of a softening filter which appearsin FIG. 18;

FIG. 22 is a schematic block diagram of a color legaliser which appearsin FIG. 11;

FIG. 23 is a schematic block diagram of a decimating processor whichappears in FIG. 11;

FIG. 24 is a schematic block diagram of a decimating filter whichappears in FIG. 23; and

FIG. 25 is a schematic block diagram of a color anti-aliasing processorwhich appears in FIG. 11.

DESCRIPTION OF PREFERRED EMBODIMENTS

A better appreciation of what is meant by the term “illegal colorpixels” may be gathered from a three-dimensional representation ofcolors which are generated with reference to the red, green and blueprimary reference colors, which is shown in FIG. 1. In FIG. 1, the red,green and blue components are represented on orthogonal axes which areperpendicular to each other. The green axis, G, is representedvertically in a scale between zero and one. The blue axis, B, isrepresented horizontally on a scale between zero and one, whereas thered axis, R, is represented as an axis going away from the plane of thedrawing, also on a scale between zero and one.

All the colors which make up a color image are generated and may berepresented by a corresponding mixture of the red, green and bluecomponents. As such, a space formed between the three axes R,G,B(RGB-space) represents these colors so that any point within theRGB-space corresponds to a particular color. Furthermore, by forming allpoints which are provided with maximum values of the red, green and bluecomponents, a cube is formed as is shown in FIG. 1. As a result, each ofthe faces of the cube corresponds to maximum values of either the red,green or blue components. For example, the plane which is parallel tothe red axis R is labelled with G=0 since this axis represents a minimumvalue of the green component for all colors lying within this plane.Correspondingly, a further parallel plane labelled G=1 corresponds toall colors for which the green component is at a maximum. Similarly theother faces of the cube are labelled with B=0 and B=1, R=0 and R=1 torepresent the minimum and maximum values that the blue and redcomponents can have within the RGB color space. The RGB-space formedwith in the cube is therefore representative of all legal colors whichcan exist in an image. The red, green and blue components of a videoimage are therefore reproduced and combined to re-generate the image,which is representative of the scene from which the image was generated.

As already explained, in order to facilitate transmission of atelevision or video image which has been generated with reference to thered, green and blue colors, the parts of the image are represented in aform other than with reference to the red, green and blue colors. Thismay be, for example, to reduce an amount of information which must betransmitted in order to represent the color video image, and thereforereduce the bandwidth of the television or video signal. One such exampleof such a form of color television and video images is the colordifference representation. In order to represent a color video image,the image is divided into a plurality of lines or rows, and each linedivided into sampling points know as pixels. As already explained, foreach pixel red, green and blue samples are generated. In order to reducethe band width of a video-image, a luminance and two chrominance signalsare generated from the red green and blue samples for each pixel. Theluminance component is generated in accordance with equation (1) below,in which the coefficients a, b, and c are selected to satisfy equation(2), and the relative ability of the human eye to detect these colors.For example the National Television Standards Committee (NTSC) colortelevision standard provided in the United States of America, definesthe following values for the coefficients a, b, and c for equation (1),a=0.587, b=0.299, and c=0.114. Similar components are provided for thePhase Alternating Line (PAL) color television system used for Europe.

Y=aG+bR+cB  (1)

a+b+c=1  (2)

U=(B−Y)=f(C _(b) −Off _(b))  (3)

 V=(R−Y)=g(C _(r) −Off _(r))  (4)

The two chrominance signals are generated in accordance with equations(3) and (4). The U component is generated by subtracting the luminancesignal sample Y from the value of blue component. The V chrominancesignal component is generated by subtracting the luminance signal samplecomponent from the red component. The C_(b) and C_(r) representationdiffer from the U and V representation of the chrominance signals byscaling factors f, g and off set factors Off_(b) and Off_(r) but areotherwise equivalent. The scaling factors f, g, are determined inaccordance with equation (2), in combination with a word length withwhich samples of the chrominance components are to be represented. As isknown to those skilled in the art, the red, green and blue signalsamples of a pixel can be recovered from the YUV or YC_(r)C_(b)components to reproduce the green component from a simple manipulationof equations (1), (2), (3) and (4). In the following description a videoimage represented as signal samples having the luminance and colordifference components of the YUV color difference signal will bereferred to as YUV color difference signals or having YUV signal format.Signal samples having red, green and blue components of the RGB spacewill be referred to as RGB color reference signals or having RGB signalformat.

Colour pixels which are represented in the form of YUV color differencesignals may be considered conceptually as falling within a colordifference signal space which is illustrated in FIG. 2. In FIG. 2 avertical axis Y is representative of the luminance component of a pixeland is represented as having a value between 16 and 235. This scalecorresponds to a range of values which can be represented in digitalform by an eight bit number. Correspondingly, the two chrominance signalcomponents are provided on the horizontal and the axis going away fromthe plane of the paper U,V. These components are represented on a scalebetween −128 and 127. This range also corresponds to a range which canbe represented with an eight bit number, but is converted to a bipolarrepresentation. An effect of representing color pixels within the colordifference space (YUV-space) is that values of the color differencecomponents YUV exist which do not correspond to points within the red,green and blue color reference space (RGB-space). This is illustrated inFIG. 2 by the cube formed from the dashed lines CL_SPACE, with each ofthe corners of the cube being labelled as black, blue, magenta, red,cyan, green, white and yellow BK, B, M, R, C, G, W, Y. As will beappreciated from the representation of the RGB-space inside the YUVcolor difference space shown in FIG. 2, an effect of transmitting colorvideo images represented as color difference signals is that signalvalues in the YUV color difference space can be generated which do notfall or correspond to those within the RGB-space.

To provide a convenient way of exchanging video and television imagesbetween independent television companies and between national televisionauthorities which otherwise use different broadcast standards fortransmitting television images, standards for representing televisionand video images were developed by the Committee Internationale desRadio Communications (CCIR). One such standard is known as the CCIR-601and provides for digital video signals to be represented as colordifference signals (YC_(r)C_(b)) in a format of 4:2:2. This 4:2:2 formatsignifies that the luminance component is represented as four sampleswith respect to each of two samples for the two chrominance signalcomponents. Colour television programmes and video images are oftenrepresented within the color difference space in digital form as for theexample CCIR-601 4:2:2 format. Within this form, video effects are oftenapplied in order to introduce special effects and color wash features asdesired by a producer of the video or television programme. This canalso have an effect of introducing values of color pixels of the videoimage represented in the YUV color difference space which do notcorrespond to values within the RGB-space. It will be understood fromthe foregoing discussion with reference to FIGS. 1 and 2, that anysample of a video or television image which produces color pixels whichdo not fall within the color reference space (RGB-space) is an illegalcolor value. With the example of representing signals within theYUV-space, it will be appreciated that when an illegal color value inthe YUV space is converted into the RGB-space and displayed, the colorvalue of this pixel of the image will not match with the surroundinglegal values, and furthermore the visual display means reproducing thevideo image will produce a color for this illegal pixel in anunpredictable way. To this end there is a requirement to detect andconvert illegal color pixels of a video image into legal color pixels.

The example embodiments of the present invention operate to convertillegal color pixels into legal pixels using any one of four methods.These four methods by which illegal color pixels can be converted intolegal color pixels will be briefly summarised in the followingparagraphs.

For the first and second methods of legalising illegal color pixels of acolor image, the illegal color pixels are legalised with reference tothe RGB-space, by converting the input signal samples in the YUV colordifference signal format into the RGB color reference format. The firstof the methods serves to limit the signal samples corresponding to thered, green and blue components of an illegal color pixel independently.That is to say that the red, green and blue components of the pixel arelimited without consideration of the effect that each component has onthe others and correspondingly the way in which the pixel is moved fromthe illegal position in the RGB color reference space to a legalposition. As a result, changing the red, green and blue signalcomponents independently has an effect of changing the picture color orhue, and the luminance and contrast.

The method of legalising by independently limiting the red, green andblue components is provided with a further improvement by converting thescale of the red, green and blue components from the scale of 0 to 1 toa bipolar scale, which is formed on an equivalently scale between −0.5and 0.5. This is implemented for an 8-bit quantised binaryrepresentation as a scale by converting the signal components from ascale between 0 and 255 to a scale between −128 to 127. This bipolarrepresentation affords a particular advantage in that the adjustmentfactors can be calculated for each signal component which can beindependently tested to confirm whether the adjustment factor will havean effect on the signal component. Furthermore, rather than simply hardlimiting, the adjustment factors, generated for bipolarised RGB signalswill have an effect of altering the signal samples even where thesesignal samples correspond to components at extreme minimum values withinthe RGB-space shown in FIG. 1, which is at zero. The bipolarised formalso provides an advantage for the interdependent RGB method in that theadjustment factors will tend to move the pixels more towards the centreof the RGB space, rather than towards zero, which represents an extrememinimum value in the unipolar form.

The method of generating the adjustment factors K_(x) for independentRGB limiting is provided in a form of pseudo code for the red componentas follows:

Rx=Rin −128;

if Rx<−128, Kr=−128/Rx;

else if Rx>127 Kr=127/Rx;

else Kr=1;

Correspondingly the pseudo code applied by a color legaliser will be asfollows:

Ry=K_(r)xR_(in);

R_(out)=R_(y)+128;

Correspondingly the same pseudo code is applied to the green and bluesignal samples.

The second of the two legalising methods which is applied to the inputsignal samples in the RGB signal format is substantially the same as theindependent method of limiting the red, green and blue componentsdescribed above. However, for the interdependent RGB limiting method,the adjustment factors Kr, Kg and Kb are first calculated, and then thelowest of these three adjustment factors is selected, and the other twofactors are set to this lowest value. The adjustment factors are thenapplied, as above. The interdependent method of legalising the red,green and blue components provides a constant hue and some changes inthe saturation and luminance of the color values. The pseudo code forgenerating the adjustment factors is presented as follows:

Kr, Kg and Kb are calculated as above;

Kmin=lowest(Kr,Kg,Kb);

Kr=Kmin;

Kg=Kmin;

Kb=Kmin.

The calculated adjustment factors are then applied to the input signalsamples Rin, Gin, Bin as above, by a color legaliser to produce thelegalised color signal samples.

The third and fourth methods of legalising color pixels are applied tothe signal samples in a form of YUV color difference signal samples.This provides an advantage in the case that the input samples arealready in the YUV color difference format since there is no requirementto provide a converter to convert the input signal samples into the formthe RGB color reference format. The first of these YUV methods is anindependent YUV method which limits each of the two chrominance colordifference components without altering the luminance component.Necessarily this results in a legalised color signal samples for thepixels in which the luminance component remains constant. Since theluminance component Y is constant, it is possible to test whether Vrepresents an illegal value by testing a corresponding R component ofthe input signals in RGB color reference space (Rin), because fromequation (4), V=R−Y. The maximum and minimum values are found inaccordance with the following pseudo code:

if (Rin>1)

set Vx=1−Y

else if (Rin<0) set Vx=−Y

else set Vx=Vin

Correspondingly for the blue chrominance component;

if (Bin>1)

set Ux=1−Y

else if (Bin<0) set Ux=−Y

else set Ux=Uin

However it has been found that this is not sufficient to ensure that thepixel is legal, because it is still possible for the green component ofa pixel to be illegal. There remains therefore a problem of ensuringthat the green component still is moved on to the either G=1 edge of thecolor reference space or G=0 edge of the color reference space.Therefore, the green component is re-calculated, which for the CCIR-601standard is performed according to equation (5):

Gx=Y−(0.299 Vx+0.114Ux)/0.587  (5)

Three adjustment factors are then calculated according to the followingpseudo code:

Ku=Ux/Uin;

Kv=Vx/Vin;

The luminance component is then evaluated to determine whether it isabove or below the centre value (0.5) in order to determine whether thegreen component should be limited to the G=0 edge of the RGB-space orthe G=1 edge of the RGB-space. According to this evaluation, anintermediate adjustment factor is calculated for the green component asfollows:

if (Y<0.5)

K_(G)=Y/(Y−G_(x)) else

K_(G)=(Y−1)/(Y−G_(x));

If however K_(G) is greater than ‘1’, which would correspond to a legalpixel, then it is set to ‘1’ so that is has no effect:

if (K_(G)>1) set K_(G)=1;

Finally the two adjustment factors for the two chrominance components Uand V are calculated by scaling each by the intermediate adjustmentfactor for the green component;

K_(uout)=Ku*K_(G);

K_(vout)=Kv*K_(G);

This method of color legalising has the advantage that the luminancecomponent remains constant, although variations in hue occur.

The second method of legalising YUV signal samples is the interdependentU V legalising method. With this method, a single adjustment factor K isestablished for both the chrominance components U, V. This adjustmentfactor is formed by calculating six intermediate adjustment factors, andforming the final adjustment factor by selecting the lowest of these sixintermediate adjustment factors (K₁, K₂, K₃, K₄, K₅, K₆). These arecalculated according to the following pseudo code:

For moving the blue component on to the blue edge where B=1;

K₁=(1−Y)/U;

For moving the blue component on to the blue edge where B=0;

K₂=−Y/U;

For moving the red component on to the red edge where R=1;

K₃=(1−Y)/V;

For moving the red component on to the red edge where R=1;

K₄=−Y)/V;

For moving the green component on to the green edge where G=1;

K₅=(Y−1)/(Y−G);

For moving the green component on to the green edge where G=0;

K₆=Y/(Y−G);

The adjustment factors for the two chrominance components U and V, areformed from the lowest of the six intermediate values:

Kuout=Kvout=lowest(K₁, K₂, K₃, K₄, K₅, K₆)

The common adjustment factor is then applied by the color legaliser tothe chrominance components of the input signal samples according to thesame pseudo code given above for the interdependent YUV method. Withthis method, the luminance and the hue of the pixel color remainconstant.

An arrangement in which a color video image maybe processed in order toconvert illegal color pixels of a color image into legal color pixels isshown in FIG. 3. In FIG. 3, an image processing apparatus 1 is shown toreceive an input signal from either a color television receiver 2, or avideo player 6 in dependence upon the configuration of a switch 3. Thetelevision receiver 2 has an antenna 4 for detecting a radio frequencysignal carrying a television image. The television receiver 2, recoversthe color television image from the detected radio frequency signal andfeeds the television image to the switch 3 via a channel 7. Also coupledto the switch 3 via a second channel 9 is a video player 6, whichprovides an example of an input signal which is generated from apre-recorded video production. The received television image and thepre-recorded video images are examples of input signals representativeof color video images which may contain illegal pixels and which may beprocessed by the image processor 1. For the example embodiment shown inFIG. 3, either of these two example input signals may be fed to theimage processing apparatus 1, by appropriately configuring the switch 3.

The image processing apparatus 1, operates to detect from the inputsignals pixels of the color images which correspond to illegal colors,and to convert these illegal colors to legal colors. The imageprocessing apparatus 1 therefore generates at an output 12 and an output14, signal samples of a legalised version of the video image received onthe input channel 5. In order to control the image processing apparatus1 and to provide a convenient user interface, a host control processor16 is provided to control the operation of the image processingapparatus 1. The control processor 16 feeds input control signals via achannel 18 and receives output signals from the image processingapparatus 1 via a channel 19. The control processor 16 is provided witha visual display unit 20, on which information received from the imageprocessing apparatus 1, conveyed by the output signals, is displayedalong with appropriate messages to an operator. The output signals arerepresentative of operating parameters and other data which indicate theposition and value of signal samples which correspond to illegal colorpixels.

The image processing apparatus shown in FIG. 3 is shown in more detailin FIG. 4 where common parts appearing in FIGS. 3 and 4 bear identicalnumerical designations. As will be appreciated from the followingexplanation, the embodiment of the present invention appearing in FIGS.3 and 4 finds application in generating a legalised color version of avideo image from input samples representative of the video image. Theinput signal samples which are representative of the pixels of the videoimage could be provided in any convenient form. If the example inputsignals from the television receiver 2, or the video player 6, were inanalogue form, then the switch 3 would include an analogue to digitalconverter to produce a digital sampled version of these analoguesignals. However, it is more likely that the television receiver 2 andthe video player 6 would produce digital signal samples representativeof the color image. Furthermore, the signal samples could representpixels of the video image in the form of color difference signal sampleswith components within the YUV-space. These signal samples may begenerated in accordance with a known standard such as the CCIR-601 4:2:2standard. Correspondingly however the input signal samples could be RGBsignal samples, or indeed the input signal samples could berepresentative of components comprising any convenient signal referencespace and therefore the example embodiments are not limited to theformat of the signal samples or the way in which the video image isrepresented. The example embodiments of the present invention will bedescribed with reference to input signal samples in the form of theCCIR-601 4:2:2 standard in which the luminance component Y isrepresented by four signal samples compared to two samples for the redchrominance signal component Cr and the blue chrominance signalcomponent Cb.

The methods of converting illegal color pixels to legal color pixelsdescribed above require that the input signal samples are in the formhaving components in either the YUV-space or the RGB-space. This isbecause the adjustment factors, designated generally as Kx are generatedfor either the YUV color difference signal format or the RGB colorreference signal format. For this reason the input signal samplesreceived via the channel 5, must be converted into either of theseforms, and in order to give an operator the option of using any of thefour color legalising methods described above, the input signal samplesare converted into both these forms.

As defined in the CCIR-601 4:2:2 standard, in order to reduce an amountof information required to represent a video image, the number ofsamples used to represent the chrominance signal components is half thatof the luminance components. This reduction in the number of signalsamples which are used to represent the chrominance components can bemade because the visual perception of the human eye to resolution of animage is less acute for color than for luminance transients. Hence theCCIR-601 4:2:2 format provides four samples for the luminance componentand two samples for each of the red and blue chrominance components.However in order to convert illegal pixels of a video image into legalcolor pixels, the video image must be provided in a form in which eachpixel has signal samples for all the components which are necessary inorder to identify a corresponding point within either the YUV-space orthe RGB-space. This can best be appreciated with reference to FIG. 5 inwhich FIG. 5A shows a representation of the color difference signalsamples in the 4:2:2 format whereas FIG. 5B provides an up sampledversion in the 4:4:4 format in which each Y luminance signal sample isprovided with an associated chrominance signal sample. In FIG. 5A, eachof the luminance signal samples of the 4:2:2 format Y₁, Y₂, Y₃, Y₄ arerepresented as each of four blocks which are arranged in a row.Associated with the First pair of luminance signal samples Y1, Y2 is afirst chrominance sample C1, and associated with the second pair ofluminance signal 10 samples Y3, Y4 is a second chrominance signal sampleC2. The two chrominance signal samples C1, C2 are arranged in a secondrow below the first row of luminance signal samples to illustrate thecorresponding association. As will be appreciated from FIG. 5A, thefirst luminance signal sample Y1 is provided with a spatial associationwith the first chrominance signal sample C1. Conceptually thiscorresponds to the pixel value within the image having a luminance valueassociated with a color value at the same position within the image.However there is no chrominance signal sample associated with the secondluminance signal sample Y2 as the chrominance signal sample C1 is notspatially associated with the luminance signal sample Y2. Similarly,whilst the second chrominance signal sample C2 is spatially associatedwith the third luminance sample Y3, there is no chrominance samplespatially associated with the fourth luminance signal sample Y4. Assuch, for those luminance signal samples Y2 and Y4 for which there is noassociated chrominance signal sample, a corresponding position withinthe RGB color reference space cannot be determined, so that anadjustment factor for these samples cannot be generated. However, bygenerating extra chrominance samples by interpolating at temporalpositions between the chrominance samples, a chrominance sample will beproduced for each of the luminance signal samples. This situation isillustrated in FIG. 5B where the extra chrominance signal samples C1 band C2 b have been generated from the original chrominance signalsamples C1 a and C2 a and which are now correspondingly associated withthe luminance signal samples Y2 and Y4 which previously did not have acorresponding chrominance signal sample. This format is known in thefollowing description as the 4:4:4 format.

FIRST EMBODIMENT

An image processing apparatus which provides a first example embodimentof the present invention is shown in FIG. 4, where parts also appearingin FIG. 3 bear identical numerical designations. In FIG. 4, the inputsamples in YCrCb 4:2:2 format are received from the channel 5 by theimage processing apparatus and fed to a color format processor 24. Thecolor format processor operates to convert the input signal samples fromYCrCb 4:2:2 format into 4:4:4 format, and to produce on connectingconductor 23 either RGB color reference signal samples or colordifference signal samples, in dependence upon control signals receivedfrom the control channel 18. The input signal samples in either RGB formor YUV form are fed to an input of an over sampling processor 26, viathe connecting conductor 23. As will be explained shortly, theover-sampling processor 26 operates to generates an over sampled versionof the input signals samples which is represented as base input signalsamples corresponding to the original sampling points of the inputsignal, and extra sampling points corresponding to the interpolated onceover sampled version of the input signal. The base input signal samplesand the extra signal samples of the over sampled version of the inputsignal are fed from two parallel conductors 25, 27 to an adjustmentfactor generator 28. Two parallel sets of adjustment factors aregenerated on two output conductors 29, 31, which respectively correspondto the base and extra input streams of samples of the over sampled inputsignal. The two parallel streams of adjustment factors are fed tocorresponding inputs of a color legalising processor 30. Also fed to thecolor legalising processor 30, on two corresponding parallel inputconductors 32, 34 are the base and extra samples of the over-sampledinput signal. After combining the adjustment factors with the inputsignal samples the legalising processor 30 presents on two outputconductors 32′, 34′ an over-sampled version of the legalised colorsignal samples for the base signal samples and the extra signal samples.The color legalising processor 30 operates to convert illegal colorpixels within the video image into legal color values by applying theadjustment factors. After being legalised, the legalised color signalsamples are fed to a decimating processor 37 which filters and decimatesto legalised signal samples to form a decimated version of the legalisedcolor signal samples which are fed to a further color legalisingprocessor 38 via a channel 89. The color legalising processor 38operates as will be explained shortly to apply a further stage of colorlegalising to the decimated version of the legalised color signalsamples so as to remove any illegal pixels which may have beenintroduced by the decimating processor 37. The decimated legalised colorsignal samples are then fed, via a channel 90 to a second color formatprocessor 39, which re-converts the legalised color signal samples intothe CCIR-601 4:2:2 standard which are fed to a signal duplicator 40. Thesignal duplicator 40 operates to duplicate the legalised color signalsamples in 4:2:2 form and to feed first and second versions of thesesignals to first and second outputs 12, 14. As shown in FIG. 3, thefirst output 12 is fed to the color visual display unit 8, fordisplaying the legalised video image, and the second output 14 is fed tothe recording device 10, for recording the legalised video image.

A schematic block diagram of the color format processor 24 whichperforms the function of converting YCrCb signal samples in the form4:2:2 into the YUV color difference version and generates a version ofthe input signal samples in RGB color reference 4:4:4 form is shown inmore detail in FIG. 6 where parts also appearing in FIG. 4 bearidentical numerical designations. In FIG. 6 the format processor 24 isprovided with a first adjustment processor 44 which receives the inputsignal samples and operates to convert these signal samples from theYCrCb format to the YUV format by providing corresponding scalingfactors in accordance with equations (3) and (4). An output of the firstadjustment processor 44 is fed to a rate conversion processor 46. Therate conversion processor 46 operates to convert the YUV signal samplesin 4:2:2 to YUV signal sample in 4:4:4 form. The YUV 4:4:4 signalsamples are fed from the rate conversion processor to a first inputterminal 47 of a controllable switch 49. The YUV 4:4:4 signals samplesare also fed to a color difference to RGB color reference conversionprocessor 48. The color conversion processor 48 operates to convert theYUV 4:4:4 format signal samples into signal samples having componentscorresponding to the RGB color reference space which are presented at anoutput of the color conversion processor 48 on a second input terminal51 of the switch 49. The controllable switch is control by a switchcontroller 45, which configures the switch 49 to feed either the inputsignal samples in either YUV form, or RGB form to the output channel 23.The switch 49 is configured in dependence upon the control signalsreceived on the control channel 18.

A better understanding of the operation of the rate conversion processor46 may be gathered from FIG. 7 which provides a more detailed schematicblock diagram of the rate conversion processor 46, for which parts alsoappearing in FIG. 6 bear identical numerical designations. In FIG. 7 theYUV 4:2:2 format signal samples are fed from an input channel 50 to asignal sample de-multiplexer 52. The signal sample de-multiplexer 52 isprovided with a switch 54 which operates to feed the signal samplesreceived from the input channel 50 to one of three output terminals 56,58, 60 under control of a switch controller 62. As will be appreciated,there are various ways in which the YUV luminance and color differencesignal samples may be multiplexed onto the input channel 50. However inaccordance with the CCIR standard 601, the 4:2:2 signal samples aremultiplex in a form in which the color difference signal samples areinterspersed with the luminance signal samples ( . . . Cr₁, Y₁, Cb₁, Y₂,Cr₃, Y₃, Cb₃, Y₄. . . ). However whatever format is used the switchcontroller 62 operates to separate the luminance and the two colordifference signal samples from each other so that the luminance signalsamples Y are fed to the first terminal 56 whereas the red and the bluechrominance signal samples U, V are fed to the second and thirdterminals 58, 60 respectively. The luminance signal samples are fed viathe first terminal 56, to a signal sample re-multiplexer 64. This isbecause the luminance signal samples are already in the required formatand the required number of samples. However, the red and bluechrominance signal samples need to be up-sampled from two samples tofour. To this end, the red and blue chrominance signal samples are fedfrom the second and third terminals 58, 60 to first and second samplingrate converters 66, 68 respectively. The sampling rate converters eachoperate to introduce extra signal samples between each of the originalchrominance signal samples so that effectively the sampling rate of thechrominance signal samples is doubled. The up-sampled chrominance signalsamples are then respectively fed to first and second half band filters70, 72, which are arranged to filter the respective chrominance signalsamples with a low pass filter having a cut off frequency substantiallyat half the new up sampled sampling rate. An effect of the half bandfilters 70, 72, is to provide interpolated signal samples at the extrasampling points introduced by the sampling rate converters 66. 68. As aresult, the red and blue chrominance signal samples are converted to aform in which the original two samples are now represented as foursamples. The four luminance samples and the four red and bluechrominance signal samples are then fed to respective terminals 74, 76,78 of the signal sample re-multiplexer 64. The signal samplere-multiplexer 64 is provided with a switch 80 which operates to connecteach of the input terminals 74, 76, 78 in accordance with the positionof the switch 80 to the output terminal 25 under the control of a switchcontroller 82. The switch controller 82 operates to present the signalsamples in a multiplexed format at the output channel 79 in accordancewith a pre-defined format. As will be appreciated any format can beconveniently used and it will be assumed in the following descriptionthat each of the four luminance and red and blue chrominance signalsamples are multiplexed sequentially together as three groups of foursignal samples (4:4:4).

Returning to the image processing apparatus 1 shown in FIG. 4, thesignal samples in either YUV or RGB 4:4:4 signal samples are fed via theconnecting channels 23 to the over sampling processor 26. As alreadyexplained the over sampling processor 26 operates to generate the oversampled version of the input signal samples which are fed via theconductors 25, 27 to the adjustment factor generator 28. The adjustmentfactor generator 28 operates in accordance with either of the abovelegalising methods to produce for each of the base and the extra inputsignal samples of each pixel of the video image within the YUV or RGBsignal space a corresponding adjustment factor designated K. In thefollowing description it is assumed that the adjustment factors K arescaling factors which are used to scale the signal samples in order toshift the position of the illegal pixels within the RGB-space so thatthe colors which they represent are legalised. The adjustment factorgenerator 28 therefore generates for each of the signal samplescorresponding to red, green and blue components within the RGB-space acorresponding adjustment factor Kr, Kg, Kb. However, if the operatorselects either of the two color legalising methods which operate on theinput signal samples in the form of YUV color difference signals, thenas indicated above, adjustment factors Ku, Kv are only calculated forthe two color difference signal components U and V. Either of the fourcolor legalising methods are selected by the operator using the controlprocessor 16, which produces the control signals representative of theselected legalising method, and which are fed to the adjustment factorgenerator 28 via the input channel 18.

The adjustment factors, which are effectively generated in over sampledform, are combined with the over sampled version of the input signalsamples, by the color legaliser 30. The over-sampled version of thelegalised color signals are then fed to the decimating processor whichproduces the legalised color signal samples which are fed to the secondcolor legaliser 38. The second color legalising processor 38 will now bedescribed in more detail with reference to FIG. 8 in which a moredetailed block diagram is provided. In FIG. 8 the second legalisingprocessor 38 is provided with a second adjustment factor generator 84 towhich the legalised color signal samples are fed. The second adjustmentfactor generator 84 also receives on a control input the control signalsfed from the control processor 16 via the connecting channel 18, whichpass through the first adjustment factor generator 28 and indicateswhich of the four color legalising methods should be applied to theinput signal samples. In accordance with which of the four methods wasselected, the second adjustment factor generator 84 operates to generatea further set of adjustment factors which are fed via a softening filter86 to a second color legaliser 88. Also fed to the second colorlegaliser 88 from a second output of the second adjustment factorgenerator 84 are the legalised color signal samples as received on theconnecting channel from the decimating processor 37. The second colorlegaliser 88 operates to combine the second further set of adjustmentfactors with the legalised color signal and generates a further versionof the legalised color signal at the output conductor 90 which is fed tothe second color format processor 39. The second color legalisingprocessor 39 therefore operates to provide a second stage of colorlegalising after the first legalised signal samples have been decimatedby the decimating processor 37. It has been discovered that whendecimating the over sampled version of the legalised color signalsamples to a version having a sampling rate corresponding to that of theinput signals which may also involve filtering the over sampled versionwith an anti-aliasing filter, some of the signal samples may be changed.As a result, a pixel which has been made legal or was already legal canbe once again made illegal. For this reason the second color legalisingprocessor 38 is provided in order to make these illegal pixels onceagain legal. It has further been discovered that an effect of applyingthe adjustment factors to the signal samples can have an effect ofexpanding the band width of the signal which can once again producedistortion in the video image and can therefore be a source ofgenerating further illegal pixels. This is illustrated by the graphicalrepresentation of the video signal shown in FIG. 9. In FIG. 9 a solidline 96 is representative of a frequency component of the video afterthe adjustment factors in an un-softened form are applied to the signalsamples representative of the video image. A dashed line 98 illustratesthat part of the video image signal 96 which was originally present inthe illegal version of the video image and is representative of illegalcolor components of the image. In the first part of the video signalrepresented by the line 96, an effect of applying the adjustment factorsun-softened is to clip or hard limit the video signal to a maximum valueshown by the flat part of the signal 97. However, an effect of this isto introduce harmonics into the video image signal which has an effectof distorting the signal and this is illustrated by the second part ofthe line 96 by the series of curves 99 which are representative of suchharmonic distortion. To substantially reduce this distortion which iscaused by applying the further adjustment factors un-softened, thefurther color legalising processor 38, is provided with the softeningfilter 86. Effectively the adjustment factor softening filter 86operates to filter the adjustment factor coefficients K to the effect oflimiting the band width of the further adjustment factor coefficientswhich adjusts the influence of the adjustment factors in dependence uponthe surrounding adjustment factors. However this is done in a way thatthe signal samples can not increase with respect to the unfilteredsamples.

As will be appreciated, the second legalising processor 38 could operatewithout the softening filter 86 and as an illustration, FIG. 10 providesan example without the softening filter 86.

SECOND EMBODIMENT

A second embodiment of the present invention will now be described withreference to FIG. 11 where parts also appearing in FIG. 3 bear the samenumerical designations. In FIG. 11, the image processing apparatus 1 isshown to receive the input signal samples via the channel 5 and as withthe first embodiment, these signal samples are in the CCIR-601 YCrCb4:2:2 format. The input signal samples are first fed to a colorconversion processor 110 which operates to scale the signal samplescorresponding to the two CrCb chrominance signals so that they areconverted to the of form of the two UV chrominance signal samples.Furthermore to allow for quantisation errors which may occur in theinput signal samples the word length with which each of the input signalsamples is represented increased from ten bits to fifteen bits. Thisprovides an increased resolution from which quantisation errors androunding effects can be detected and processed, so that these errors canbe avoided. Coupled to a connecting channel 112 from the colorconversion processor 110 is a rate conversion processor 114. The rateconversion processor 114 operates to convert the input signal samples in4:2:2 format into 4:4:4 format substantially in accordance with the rateconversion processor 46 shown in FIG. 6 with the accompanyingexplanation as presented for the first embodiment. Therefore presentedon an output channel 116 are the input signal samples as received fromthe input channel 5 but in a form of YUV 4:4:4 format. In FIG. 11, eachof the channels between the respective parts is representative of colordifference signal samples YUV in 4:4:4 format unless otherwise stated.The signal samples are fed from the connecting channel 116 to an oversampling signal processor 118. The over sampling signal processor 118operates to generate a four times over sampled version of the YUV 4:4:4signal samples and presents these signal samples on four parallel outputchannels 120, 122, 124, 126 in 4:4:4 format. Thus on the first outputchannel 120, the original input signal samples are provided whereas onthe second output channel 122, a first over sampled version of the inputsignal samples is generated. The signal samples provided on the secondoutput channel 122 therefore correspond to the first over sampledversion that is by increasing the sampling rate by a factor of two. Onthe third and fourth output channels 124, 126 further extra signalsamples are generated in accordance with a further doubling of thesampling rate. Further explanation of the operation of,the over samplingprocessor 118 will be provided shortly. However in the followingdescription, those signal samples which are associated or correspond tothe sampling points of the original input signal samples beforeup-sampling) will be referred to as base samples, whereas those signalsamples which correspond to temporal points which are added afterover-sampling will be referred to as extra samples.

Each of the outputs from the over sampling processor 118 is fed to acolor reference converter 128. The color reference converter 128operates to produce for each of the YUV signal samples received on thefour parallel channels 120, 122, 124, 126, an equivalent version in theform of RGB color reference signal samples in 4:4:4 format in parallelwith the YUV format. Thus, the RGB color reference converter 128provides at each of four pairs of output channels 130, 132, 134, 136,138, 140, 142, 144, respectively YUV and RGB versions in 4:4:4 formatwith each pair corresponding to the base and respective extra signalsamples generated by over sampling. This is to provide the input signalsamples in both of the two formats (YUV or RGB) from which either of thefour legalising methods may be applied.

As shown in FIG. 11 an adjustment factor processor 146 operates toreceive the over sampled version of the input signal samples in YUV andRGB formats in parallel for each of the base input signal samples andthree corresponding over sampled versions associated with each of thebase signal samples. Correspondingly, therefore the adjustment factorprocessor 146 operates to generate associated adjustment factorsaccording to either of the two YUV color difference legalising methodsor the two RGB color reference legalising methods. In the former caseonly two sets of adjustment factors are generated for each pixel for thesignal samples one corresponding to the red color difference componentsU, and the other corresponding to the blue color difference componentsV. In the latter case a set of adjustment factors is generated for eachof the red, green and blue components of the RGB-signal format.

The adjustment factors are generated for each of the base and each ofthe extra signal samples associated with the base samples. These areprovided at four corresponding outputs 148, 150, 152 and 154. Each ofthe four parallel versions of the adjustment factors are then fed to anadjustment factor softener 156 which operates to soften these adjustmentfactors by adapting and changing the adjustment factors in order toeffect a reduction in distortion which is produced by applying theadjustment factors without softening. In association with the softeningprocess which will be described shortly, the adjustment factors aredecimated to the effect that the four times over sampled version of theadjustment factors is reduced to a twice over-sampled version. For thisreason only two output channels 158. 160 are provided to feed thesoftened adjustment factors to a color legaliser 162, on the two outputchannels 158, 162 associated respectively with the base signal samplesand the extra signal samples corresponding to a twice over sampledversion.

As shown in the bottom half of FIG. 11, the color legaliser 162 receivesthe over sampled version of the adjustment factors on the input channels158, 160. Also fed to the color legaliser 162 on two pairs of furtherinput channels 164, 166, 168, 170, are the YUV and RGB 4:4:4 versions ofthe input signal samples generated by the over-sampling processor 128 atthe first and second input channel pairs 130, 132, 134, 136 which werealso fed to the adjustment factor processor 146. In accordance with themethod chosen for legalising the video image, the color legaliser 162operates to combine the adjustment factors with the version of the inputsignal samples received on the input channels 164, 166, 168, 170 inorder to produce at respective output channels 172, 174 legalised colorsignal samples. The first output channel 172 provides legalised colorsignal samples corresponding to the base input signal samples whereasthe second output channel 174 provides legalised color signal samplescorresponding to the extra over sampled version of the input signalsamples. Thus in effect, the output channels 172, 174 provide an oversampled version of legalised color signal samples corresponding to theinput signal samples but over sampled at twice the rate of the inputsignal samples, which is at rate 8:8:8. Also generated at a furtheroutput channel 176 are data representative of a plurality of modifiedflags, each of which is associated with one of the samples within thebase and the extra legalised color signal samples to indicate whetherthis signal sample in the legalised form has changed with respect to thecorresponding base and extra input signal samples.

The over sampled version of the legalised color signal samples are fedfrom the output channels 172, 174 to a decimating processor 178. Thedecimating processor filters 178 filters and decimates the legalisedcolor signal samples from a rate 8:8:8 to a rate 4:4:4. After filteringand decimating the over sampled version of the legalised color signalsamples, the decimating processor 178 feeds the filtered legalised colorsignal samples to a second color reference converter 182 via an outputchannel 184. The modified flags are fed in parallel to the decimatingprocessor 178 via a channel 176, and are further passed directly to acolor anti aliasing processor 180 via a further control channel 188. Ina case where either of the two RGB color legalising methods wereselected, the second color reference converter 182 will operate toconvert the legalised color signal samples received on the input channel184 into YUV color difference signal samples. If either of the two YUVlegalising methods were used, then legalised color signal samples willalready be in the YUV form. Therefore the second color referenceconverter 182 will provide the legalised color signal samples in YUVform correspondingly on an output channel 190 connected to the coloranti aliasing processor 180. The color anti aliasing processor 180 thenreceives the legalised color signal samples in 4:4:4 rate in the form ofthe YUV color difference samples. The color anti aliasing processor 180also receives on the control channel 188 the data representative of themodified flags.

The color anti aliasing processor 180 operates to filter the oversampled chrominance components of the UV of the color difference, inpreparation for decimating the chrominance components from four samplesto two samples. However the legalised color signal samples produced atan output channel 194 remain in the over sampled format at a rate 4:4:4.

The legalised color signal samples in YUV 4:4:4 format are fed from thecolor anti aliasing processor 180 via the output channel 194 to a secondadjustment factor generator 196. The second adjustment factor generator196 operates to generate a further set of adjustment factors usingeither of the two methods described above for generating adjustmentfactors for YUV signal samples. These further adjustment factors are fedon a first output channel 198 to a second adjustment factor softener200. The down sampled version of the legalised color signal samples isthen output on a second channel 202 and fed to a second color legaliser204. After passing through the second color softener 200, the furtheradjustment factors are fed to the second color legaliser 204 from anoutput channel of the second adjustment factor softener 206. The secondcolor legaliser 204 operates to combine the softened further adjustmentfactors with the down sampled version of the legalised color signalsamples and generates a final version of the legalised color signalsamples which represents a version of the video image having legalcolors. The final version of the legalised color signal samples is thenfed to a second color conversion processor 208 via a connecting channel210. The second color conversion processor 208 operates to scale thesignal samples in YUV format so that the chrominance signals arere-converted into YCrCb form color difference signal samples. The secondcolor conversion processor 208 also operates to decimate the chrominancesignal samples so that the legalised color signal samples in 4:2:2format are presented at an output channel 211. Finally a signalduplicator 209 generates a second version of the legalised color signalsamples, and the first and second versions are presented on the twooutput channels 12, 14. The output channel 19, provides informationcorresponding to the modified flags which represents the position andvalue of illegal pixel components which are superimposed by the controlprocessor 16 on the displayed version of the video image represented bythe legalised color signal samples.

A better appreciation and understanding of the operations of the secondembodiment of the present invention will now be provided with referenceto further diagrams which illustrate the operation of each of theprocessors within the image processing apparatus shown in FIG. 11 inmore detail. In order to reduce repetition, those features whichgenerally correspond with those of the first embodiment will not bedescribed in detail. Therefore, for example the operation of the rateconversion processor 110 and color conversion processor 114 issubstantially as described for the color conversion processor 44 andrate conversion processor 46 presented in FIGS. 5 and 6 of the firstembodiment. However, the over sampling processor 118 will now bedescribed in more detail with reference to FIGS. 12, 13 and 14 whereparts also appearing in FIG. 11 bear the same numerical designations.

In FIG. 12, the over sampling processor 118 is shown to receive thecolor difference signal samples in YUV 4:4:4 format at a de-multiplexer220. The de-multiplexer 220 is provided with a switch 222 which operatesunder control of a switch controller 224 to switch the samples receivedon the input channel 116 to each of three terminals 226, 228, 230. Eachof the terminals 226, 228, 230 is coupled respectively to one of threeup sampling processors 232, 234, 236. The de-multiplexer 220 operates toseparate the color difference signal samples by feeding respectivelyeach of the luminance and red and blue chrominance signal samples to arespective one of the three up sampling processors 232, 234, 236. The upsampling processors each operate to generate a four times over sampledversion of the signal samples received from the de-multiplexer 220. Anup sampling processor is shown in more detail in FIG. 13. The upsampling processor 232 is provided with a first sample rate converter238 which operates to double the sampling rate of the signal samplesreceived via the channel 231 by introducing sampling points between thereceived base signal samples. The signal samples which are extra signalsamples at these new sampling points are then generated by feeding theconverted luminance signal samples to a half band filter 240 whichoperates to filter the converted luminance signal samples with a lowpass filter having a cut off frequency substantially equal to half thenew sampling frequency of the signal samples. An effect of this is toproduce at the output of the half band filter 241, a version of theluminance signal which comprises the base signal samples (S₀) and extrasignal samples (S_(i)) which therefore corresponds to a twice oversampled version of the luminance signal samples. The output of the firsthalf band filter 240 is then fed to a second up sampled rate converter242 which operates to once again introduce sampling points between theexisting signal samples so that when the rate converted signal is fed toa second half band filter 244 a four times over sampled version of theluminance signal samples is generated at an output channel 245. Theresult of this process can be appreciated from the graphicalrepresentation of the signal samples presented in FIG. 14. In FIG. 14the original or base luminance signal samples are designated S₀ whereasthe first extra signal samples generated by the first over samplingstage are designated S_(i). The third and fourth signal samples added bythe second over sampling stage are designated S_(ii1) and S_(ii2). Ineffect there is provided a four times over sampled version of theoriginal signal samples. This is not only applied to the luminancesignal samples but also the U and V chrominance signal samples by thefurther over sampling processors 234, 236.

The over sampling processor 232 further operates to re-multiplex theover sampled version of the signal samples by feeding the signal samplesat the output of the second half band filter 245 to a controllableswitch 246 which operates under influence of a switch controller 248 tofeed the signal samples successively to each of four first in first out(FIFO) buffers 250, 252, 254 and 256. The switch controller 248therefore operates to separate the base signal samples S₀ which are fedto a first output terminal 258, the first over sampled extra signalsamples S_(i) which are fed to a second output terminal 260, and thethird and fourth extra signal samples S_(ii1), S_(ii2) which are fed tosubsequent output terminals 262, 264. Each of these signal samples arefed to the FIFO buffers which operate, to group these signal samples ingroups of four and then present correspondingly in parallel the base andcorresponding extra signal samples on respective parallel outputchannels 266, 268, 270, 272. The control of the FIFO buffers is providedby a control circuit 274 which is coupled to each of the FIFO buffers bya control channel 276. In effect therefore at the output of each of theover sampling processors 232, 234, 236 for each of the three componentsof the YUV color difference signals there is presented four versions foreach signal sample received on the input channels 231, 233, 235. Theoutput from each of the up-sampling processors are then cross-connectedto one of four multiplexing circuits 278, 280, 282, 284. Thecross-connection is afforded by a set of channels 286 which operates tofeed each of the base and the extra signal samples to the respectivemultiplexers 278, 280, 282, 284 which multiplex each of the colordifference signal components YUV on to an output channel. As a resultthe four parallel output channels 120, 122, 124, 126 provide in parallelthe base and the over sampled versions of the signal samples in thecolor difference YUV 4:4:4 format. Thus, for example the multiplexer 278is provided with a switch 236 which is controlled by a switch controller288. The switch controller operates to select the four samples fed fromthe FIFO buffers to each of the input terminals 288, 290, 292 so thatthey are multiplexed on to the output channels 220, 222, 224, 226. Itwill be understood by those skilled in the art that there are other waysof forming the over sampled version of the input signal, which may notrequire rate conversion as described above. For example, in a preferredembodiment the three extra samples corresponding to the over sampledsignal are generated by copying each of the respective base input signalsamples, and representing the corresponding time shifts for each ofthese samples to generate the four parallel streams of samples.

As will be appreciated from the above description of the parts andoperation of the over sampling processor 118, a four times over sampledversion of the color difference signal samples is generated from whichthe corresponding adjustment factors are produced. Distortion in a videoimage is produced when the legalising methods are applied as a result ofinformation within the video image not being adequately represented bycolor difference signal samples having a sampling rate corresponding to4:4:4. This is because the video image in the analogue domain, oftencontains frequency components which are higher than those which can berepresented at the lower sampling rate of 4:4:4 so that when colorlegalising is performed, and the adjustment factors are applied to thesignal samples of the color video image, further illegal color pixelsare produced within the video image as a result of distortion caused byaliasing errors as a result of such higher frequency components notbeing adequately represented at this lower sampling rate. Furthermore,unless the temporal sampling points of the video image coincide with themaximum values of the analogue video image signal, then these maximumvalues will also not be represented by the sampled digital version ofthe video image so that even after the adjustment factors are applied tothe video image in the digital domain, illegal color values will stillbe present in the analogue version of the video image. It is for thisreason that the color image is super-sampled, by representing the videoimage effectively at a rate of 16:16:16 for the YUV color differencesignal format, by generating for the base signals samples having a4:4:4-rate the three extra signal samples also having a YUV format4:4:4. Of course, if it were possible then the video signal should berepresented by as a continuous signal having an infinite sampling rate.However, it has been found that a benefit resulting from furtherincreasing the sampling rate above a sampling rate corresponding to rate16:16:16 does not bring a corresponding increase in performance byfurther reducing distortion with regard to physical problems inimplementing a higher sampling rate. It is in this over sampled formatthat the adjustment factors are generated and applied to the version ofthe input signal samples which has an effect of substantially reducingdistortion associated with an inadequate sampling rate and thereforeprovides an improvement in making legal the colors of the video image.

After the over sampled version of the input signal samples in the YUVsignal format are generated by the over sampling processor 118, aversion of the YUV signal samples are generated in the form of RGBreference signal samples having components with reference to the RGBcolor reference space. This is effected by the color reference converter128 which is shown in more detail in FIG. 15. In FIG. 15 the colorreference converter 128 is shown to include for each of the fourversions of the over sampled YUV color difference input signals one offour color converters 280 which is coupled to a respective one of theoutput channels 120, 122, 124, 126, from the over sampling processor118. Each of the color converters operates to convert the colordifference signal samples YUV into corresponding RGB color referencesignal samples. The RGB color reference signal samples are presented atrespective output channels which are then connected to respective RGBshift processors 282. The RGB shift processors 282 operate to convertthe RGB color reference signal samples each of which has a value between0 and 255 of the eight bit samples into a bipolarised version of the RGBsignal samples which fall on to an equivalent scale between −128 and127. The bipolarised RGB color reference samples are presented at anoutput to the RGB shift processors on the output channels 132, 136, 140,144. In parallel the equivalent YUV color difference signal samples arecoupled directly from the four parallel connecting channels 120, 122,124, 126 from the over sampling processor directly to the correspondingoutput channels 130, 134, 138, 142 of the color reference converter 128.

As already explained, a particular advantage provided by the RGB shiftprocessors 282, which form part of the color reference converter 128shown in FIG. 15 which can be more easily understood from a graphicalrepresentation of the RGB-space in unipolar and bipolar form shown inFIG. 16. In FIG. 16a, the three reference axes of the red green and bluecolors are shown as they appeared in FIG. 1. but scaled between 0 and255. Shown in FIG. 16b is a corresponding representation in the bipolarformat generated by the RGB shift processor 282. In FIG. 16b each of thered, green and blue color axes are now scaled so that the two extremesof the scale lie between values of −128 and 127 with the centre of theaxis being at zero. As a result, when the adjustment factors K aregenerated by the adjustment factor processor 146, and applied to theinput signal samples in RGB color reference form, the effect will be toshift the corresponding pixels within the RGB- space. For theinterdependent RGB legalising method this will shift illegal pixels moretowards the centre of the space rather than to the extreme minimum ofzero which would be the case if the adjustment factors were generatedfor the color reference axes shown in FIG. 16a. Furthermore in a casewhere an illegal color lies close to or in the vicinity of a minimumcolor value, that is at or near ‘0’ in FIG. 16a, an effect of scalingwith an adjustment factor would be essentially to multiply the signalsample by zero in order to move the corresponding point within the colorreference space to the minimum value for this particular axis. Thisrepresents more of a hard limit and so by representing the color signalsamples in the RGB reference space on a bipolarised form as shown inFIG. 16b, the influence of the adjustment factors will be moreaccurately represented. Furthermore the softening process when appliedto the adjustment factors will be more effective.

Returning to FIG. 11, the adjustment factor processor 146 operates togenerate the adjustment factors in accordance with the four methods oflegalising the illegal color pixels of a video image as alreadydescribed. However in the second embodiment this is performed for eachof the four sets of YUV 4:4:4 color difference signal samples generatedby the over-sampling processor 118. The adjustment factor processor 146according to the second embodiment of the present invention is shown inmore detail in FIG. 17 where parts also appearing in FIG. 11 have thesame designations. The adjustment factor processor 146 is shown to havean adjustment factor generator 284 for each of the four pairs of inputchannels 130, 132, 134, 136, 138, 140, 142, 144. The adjustment factorprocessor 146 also receives on the input channel 18 control signalswhich are fed to a control processor 286. The control processor 286generates appropriate control commands which are fed to each of theadjustment factor generators 284 via a control signal channel 288. Thecontrol signals serve to indicate to the adjustment factor generatorswhich of the four color legalising methods should be used. Thus, inaccordance with either of the above mentioned legalising methods forconverting illegal color signal samples in either the YUV colordifference signal format or the RGB color reference signal format theadjustment factor generators 284 operate to generate an adjustmentfactor for each of the input signal samples having either YUV colordifference components or in the equivalent form of the RGB colorreference components. The adjustment factors are fed on a first outputchannel 290 to a quantisation processor 292. On a second output channel291, the input signal samples are fed to a second input of thequantisation processor 292, in the form of the RGB color referencesignal samples.

The adjustment factor quantisation processors 292 provide a furtherimprovement to the color legalising process, by reducing a possibilityof pixels within the color image becoming illegal, or remaining illegalas a result of quantisation and rounding errors. This improvement isprovided by comparing each of the input signal samples in the form ofRGB color reference signal samples with a quantisation threshold. Asalready explained, the word length of the digital samples with which theinput signals samples are represented is increased by the colorconversion processor from ten bits to fifteen bits. Correspondingly,therefore the resolution to which the input signal samples can berepresented has increased, thereby allowing the quantisation of theoriginal samples to be investigated with respect to a quantisationthreshold, which is determined with respect to the original word length.If the signal sample is less than the quantisation threshold then theadjustment factor calculated for this input signal sample is set to avalue of ‘1’ which therefore has no effect when multiplied with thecorresponding input signal sample. The reason for introducing thiscomparison with the quantisation threshold is that the adjustmentfactors and the input color signal samples are only represented towithin a finite quantisation level. For the present embodiment each ofthe original input signal samples are only represented with ten bitsamples. The quantisation errors can cause the adjustment factors tochange an otherwise legal pixel of the color image into an illegalpixel, because in the ten bit form a signal sample may be rounded up toa value which makes it appear as illegal. The quantisation error iscalculated by processing a signal sample from the input channel 5 with avalue of half the least significant bit of the ten bit representation.The final value of the signal sample presented at the output of theadjustment factor generator 284 provides a measure of the quantisationerror q. The quantisation error results in an adjustment factor beinggenerated which indicates that an input signal sample should belegalised whereas in fact it should not. By setting the adjustmentfactors for a corresponding input signal sample having a value whichfalls between the value indicated by equation (6) below, to a value ofone, an improvement is provided in reducing a chance of illegal pixelsbeing generated or retained in the color image.

1<S _(in)≧1|q|  (6)

where in equation (6), S_(in) is either of the red, green and bluecomponents of a pixel. The pseudo code, for the input signal samples inRGB bipolar form, where the R, G, and B values have already had 0.5subtracted from them and which corresponds to the process performed bythe quantisation processor 292 is presented as follows:

if ((|R|-RSlack)<=0.5)&(|G|-GSlack)<=0.5)&(|B|-BSlack)<=0.5))

set K=1.00

else

set K=Kmin

where Kmin is calculated as before by the color legalising method, andRSlack, GSlack, BSlack are predetermined values corresponding to thequantisation threshold q, and are calculated by feeding a signal samplehaving a value of half the least significant bit of the ten bit wordused to represent the signal samples.

In order to further improve the accuracy with which the adjustmentfactors are generated, the quantisation threshold RSlack, GSlack, Bslack(|q|) is generated with reference to the RGB color reference space, asthis is the signal space which was used to generate the color image andwill be used to reproduce the color video image. This providesconsistency in applying the quantisation threshold, and is fed by thecontrol processor 286 to the quantisation processor. Thus regardless ofthe form in which the input signal samples are processed, thequantisation threshold is set in accordance with the RGB color referencespace, so that even if the adjustment factors are calculated for theinput signal samples in the form of YUV color difference signal samples,the RGB signal samples are tested with respect to the quantisationthresholds RSlack, GSlack, Bslack. This is because there is a differencebetween the quantisation factor produced when adjustment factors aregenerated for the YUV color difference signal samples and RGB colorreference signal samples. Furthermore, because the input signal has beenover sampled, and the extra signal samples effectively represent samplesof further resolution of less importance than the original samples, thequantisation threshold q can be set to different amounts in dependenceupon the relative importance of the base and extra samples.

After any quantisation errors have been removed by the quantisationprocessor 292, the resulting output signal is fed respectively from eachof the quantisation processors 292 to an adjustment factor biasingprocessor 294 via respective channels 296. The adjustment factorprocessor 146 is provided with a further advantage by arranging for thebiasing processors 294 to introduce a pre-biasing constant into each ofthe adjustment factors under control of the control processor 286. Ithas been found that when the over sampled legalised version of the colorsignal samples are filtered with an anti-aliasing filter and decimated,some of the previously legalised pixels of the video image can againbecome illegal and other legal parts of the image can produce illegalcolor pixels, as a result of the signal samples being changed during thefiltering and decimation processes. In order to reduce the possibilityof legal color pixels becoming illegal, the biasing processor 294operates to scale each of the adjustment factors with a biasing constantas shown generally by equation (7) where K′_(x)is the adjustment factorbefore biasing.

K _(x) =K′ _(x)α_(g)  (7)

By making the biasing constant ag greater than 1, the scaling factorsare correspondingly increased closer to 1, so that their effect on theinput signal samples is reduced. As a result, the effect of legalisingthe video image is reduced so that if desired, the video image canremain proportionally more illegal. If however, the biasing constantα_(g) is less than 1, then the effect of the adjustment factors isincreased so that the possibility of legalised color values becomingonce again illegal is proportionally reduced. This has an equivalenteffect of shrinking the RGB color reference space. This is illustratedin FIG. 2 by the second cube CL_SPACE having a broken line within thefirst color reference cube of the YUV color difference reference space.The biasing constant is applied after the quantisation errors have beenremoved by the quantisation processor. However when combining thequantisation processor with the biasing constant, regard must be had toa combined effect on making otherwise legal pixels illegal, for both theYUV color difference signal and the RGB color reference signals. Inpseudo code the combined effect of the quantisation processor and thebiasing processor is as follows:

If ((|R|<=0.5)&(|G|<=0.5)&(|B|<=0.5))

set K=Kminα_(g)

else

if ((|R|-RSlack)<=0.5)&(|G|-GSlack)<=0.5)&(|B|-BSlack)<=0.5))

set K=α_(g)

else

set K=Kminα_(g)

The biased adjustment factors are presented on the four parallel outputchannels 148, 150, 152, 154 and fed to the adjustment factor softener156. As with the input to the adjustment factor processor 146, each ofthe four outputs is correspondingly associated with one of the base andthe three extra signal samples of the over sampled version of the colorinput signal samples provided by the over sampling processor 118.Correspondingly, therefore each of the outputs shown in FIG. 17 isprovided with corresponding adjustment factor values K₀, K_(i), K_(ii1)and K_(ii2). The adjustment factor softener is shown in more detail inFIG. 18 where parts also appearing in FIG. 11 bear the same numericaldesignations. The adjustment factor softener 156 shown in FIG. 18 isprovided with a de-multiplexing processor 300 to which each of the fourinput channels 148, 150, 152, 154 are fed. The de-multiplexing processor300 operates to separate the components of the adjustment factorsassociated with either the two chrominance signal components of the YUVcolor difference signal space or the red green and blue signalcomponents of the color reference space depending on which of the fourcolor legalising methods is being employed. The de-multiplexer 300operates to feed each of the separated signal components and to each oftwo associated softening processors 302 via channels 304, 305. A betterunderstanding of the operation of the de-multiplexer 300 is providedfrom the diagram in FIG. 19 where parts also appearing in FIG. 18 bearidentical numerical designations.

As shown in FIG. 19 the signal samples received on the four inputchannels 148, 150, 152, 154 are fed respectively to one of fourseparating processors 306. The separating processors 306 operatesubstantially in accordance with the signal separating processor 220shown in FIG. 12 and so further explanation will not be repeated.However, in effect the separating processors 306 separate the threecomponents of the adjustment factors which were generated with respectto the corresponding RGB signal components, and feeds each of theadjustment factors associated with each of these signal components toone or three corresponding multiplexers 310 via channels 308. In thecase where the adjustment factors were generated with respect to the YUVcolor difference signal components, only two of the multiplexers 310 arerequired for the adjustment factors generated with respect to the U andV color difference components. The multiplexers 310 are provided withfour input terminals which are connected to respective outputs from thesignal separating processors 306. Each of the input terminals 312, 314,316, 318 is connected by a switch 322 in turn to the output terminal 304under control of a switch controller 320. A second version of theadjustment factors for each signal component is provided at a second setof output channels 305 via a two-stage delay circuit. The adjustmentfactors produced at the first and second output channel pairs 304, 305,provide a serial stream of adjustment factors associated with each ofthe signal components of the RGB signal space or YUV signal space, withthe serial stream from the second output 305 being delayed with respectto the first output 304 by two signal samples. The effect of thede-multiplexer 300 can be appreciated from the graphical representationof the adjustment factors which are produced at the pairs of outputchannels 304, 305 shown in FIG. 20. In FIG. 20 the signal samples fromthe first output 304 are shown in FIG. 20a and the second output 305 areshown in FIG. 20b. The signal samples are represented as amplitude withrespect to a time. Associated with each of the signal samples is thecorresponding designation as to whether the adjustment factor isassociated with one of the base signal samples K₀ or the first oversampled signal samples K_(i) or the second and third adjustment factorsamples Kii1 and Kii2 provided from the second over sampling of theinput signal sample.

For each of the adjustment factors associated with each of therespective YUV or RGB signal components, produced at the outputs of thede-multiplexing processor 300, there is provided a softening filter 302.Thus, each pair of the first and second outputs 304, 305 is providedwith a corresponding pair of softening filters 302. The softening filter302 according to the second embodiment of the present invention is shownin more detail in FIG. 21 where parts also appearing in FIG. 18 have thesame numerical designations. The softening filter 302 shown in FIG. 21is provided with an inverter 324 which operates to reverse polarise eachof the adjustment factors received on the input channel 304. The outputof the inverter 324 is connected to a first input of an adder 326, andto a second input to the adder 326, a value of 1 is connected. As aresult, the adjustment factors at the input to the softening filter 302,are converted to an inverted scale, that is from one to zero, to zero toone. The inverted adjustment factors are then fed to a shift registerwhich is shown in FIG. 21 to be comprised of nine stages 330 which areinterconnected with delay elements 332. A central tap 334 of the shiftregister 328 is connected directly to a final non-additive mixing stage336. At respective corresponding stages either side of the central tap334, the stages of the shift register are paired and channels from eachof these paired stages connect the output of these stages to first andsecond inputs of intermediate non-additive mixers 338. Each of theintermediate non-additive mixers operates to select the greater of thetwo adjustment factors received from the first and second inputscorresponding to the paired outputs from the stages of the shiftregister 328. The selected greater of the two inputs is fed to the finalnon-additive mixing stage 336 via a multiplier 340. To a further inputof the multipliers 340 a scaling coefficient W_(n) is applied whichoperates to scale the greater sample produced by the intermediate.non-additive mixers 338 before being applied to the final non-additivemixing stage 336. The final non/additive mixing stage 336 operates tocompare each of the adjustment factors received on the five inputs andto select the lesser of the five inputs as an output adjustment factor.This selected adjustment factor is fed from the output of the non-linearprocessing stage 336 to the input of a second inverter 344 whichoperates to reverse polarise the adjustment factor. This is applied to afirst input of an adder 346 and a value of one is applied to the secondinput of the adder 346 so that at the output 348 of the softening filter302, the selected adjustment factor is once again inverted from thescale of one to zero, to zero to one.

As will be appreciated from the operation from the softening filter 302described with reference to FIG. 21, for each adjustment factorpresented and fed to the input 304, a softened adjustment factor isgenerated at the output 348 from a selection of this adjustment factorin combination with the other previously received adjustment factorsstored in the shift register 328. The adjustment factors are invertedthat is to say reverse scaled from one to zero, to zero to one beforebeing applied to the shift register in order to provide an advantageouseffect by which those adjustment factors which are closest to one andwould therefore have least effect when scaled with the correspondinginput signal samples have least value in the selection process providedby the softening filter. Correspondingly those adjustment factors whichwill have most effect when scaled with the input signal samples that isto say those closest to zero will be inverted to be closest to one andtherefore have most influence within the softener. As a result, thesoftening process produced by the softening filter is more stronglyapplied to those adjustment factors which have greatest effect on theinput signal samples particularly having regard to quantisation errorsand other inaccuracies in the quantised values of the adjustment factorsand signal samples. Furthermore by applying a window function, which isprovided by scaling each of the inputs to the final non-linearprocessing stage 336 using the multipliers 340 with the coefficientsW_(n) of the window function, a shaping window represented by thecoefficients is convolved with adjustment factors. This reduces aliasingerrors in the filter output 302. Although the number of multipliers isshow to be four, which effectively corresponds to a five tape windowfunction, it will be appreciated that the window can be of any length.

As will be appreciated from the above explanation, each of the softeningfilters 302 is applied respectively to the adjustment factors associatedwith each of the components of the signal samples of the color inputsignals. Furthermore, the same softening filters are also applied to thesecond output 305 from the de-multiplexer 300 which provides adjustmentfactors which are delayed in time by two samples. An effect of thissecond softening filter applied to the second output 305 is to produce asecond softened adjustment factor to be applied to the extra inputsignal samples. The outputs from each of the softening filters 302 arefed from output channels 348, 350 to first and second multiplexers 352,354. The multiplexers form the adjustment factors into two streams eachof which has adjustment factors associated with each of the signalcomponents in either the 4:4:4 color difference signal format or the RGB4:4:4 color reference format. Each stream associated with base and extrasamples is provided at the associated output channels 158, 160. Thus,effectively the adjustment factor softener 156 not only softens theadjustment factors by adapting their value in accordance with asmoothing or band limiting effect but also decimates the adjustmentfactors by halving the number of adjustment factors, so that theadjustment factors are now in an over sampled form of 8:8:8 rate.

As will be appreciated from viewing the diagram of the image processingapparatus 1 shown in FIG. 11, the image processing apparatus 1 generallyforms into two parallel streams of processors, the first upper streambeing associated with the task of generating the softened adjustmentfactors and the second lower parallel stream being provided to apply theadjustment factors. To this end, the color legaliser 162 receives thesoftened adjustment factors from the two output channels from thesoftener 156, at first and second input channels 158, 160, and at twofurther pairs of inputs 164, 166, 168, 170, the YUV 4:4:4 colordifference signal samples and the color RGB 4:4:4 color reference signalsamples are fed on pairs of channels for the over sampled version of theinput signal samples generated correspondingly at the output of thecolor reference converter 128. Thus the pairs of signal samples eachproduce a version of the base input signal samples and a versioncorresponding to the extra input signal samples produced from oversampling the input signal. The color legaliser 162 is shown in moredetail in FIG. 22 where parts also appearing in FIG. 11 bear the samenumerical designations.

In FIG. 22 the color legaliser 162 is shown to receive signals from thecontrol signal bus 149, which are fed to a first control processor 356.The control processor 356 operates to feed a control signal indicativeof which of the four methods for legalising illegal color signal sampleshave been selected, via a control channel 358 to an input of each of twomultiplying processors 360. The multiplying processors 360 also receivethe two pairs of input channels 164, 166, 168, 170 which feedrespectively the base and the extra signal samples produced from theover sampling processor 118 for the YUV color difference input signalsamples or the RGB color reference input signal samples to themultiplying processors 360. In accordance with which of the four methodsfor legalising the input signal samples selected by an operator andindicated by the control signals fed from the control processor 356, themultiplying processors 360 operate to multiply the softened adjustmentfactors received for the base and the extra input signal samples on thefirst and second input channels 158, 160 by the corresponding componentsof the input signal samples in either the YUV color difference form orthe RGB color reference form. At the output of each of the multiplyingprocessors 360, legalised color signal samples are produced, which arereceived at polarising converters 361. The color polarising converters361, operate to convert legalised signal samples which are produced frominput signal samples in the RGB bipolar form to into a unipolar form.This is to reverse the operation of the color bipolariser 282 for signalsamples produced on a first output 172, for base legalised signalsamples and the output channel 174 for the extra legalised signalsamples.

The first and second inputs from the adjustment factor softener are alsofed to a second control processor 362. The second control processor 362operates to determine for each of the received adjustment factorscorresponding to the base input samples and the extra input samples,whether the corresponding adjustment factors have had an effect ofchanging the corresponding version of the input signal samples in thelegalised color signal samples. If the adjustment factors have an effectof changing the input signal sample with respect to the correspondinglegalised color signal sample then a flag is set and generated at theoutput 176 of the second control processor 362 to indicate this fact. Asdescribed in the next paragraph, these flags, known as modified flags,will be used in the decimating processor 178 and the color anti-aliasingprocessor 180.

The legalised signal samples are fed from the first and second outputchannels 172, 174 to the decimating processor 178, which is shown inmore detail in FIG. 23, for which parts also appearing in FIG. 11 bearthe same numerical designations. The legalised color signal samplescorresponding to the base input signal samples, received on the firstinput 172, are fed to a first input of a de-multiplexer 364. The extralegalised color signal samples corresponding to an over sampled versionof the legalised color signal are fed from the connecting channel 174 toa second input of the de-multiplexer 364. The de-multiplexer 364operates substantially in accordance with the de-multiplexer 300,previously described for the adjustment factor softener in FIG. 18,except that the de-multiplexer 364 only requires two inputs andtherefore correspondingly will only have two signal separators. Thus, aserial version of the signal samples corresponding to each of the YUV orRGB components in over sampled form are separated into each of theirthree respective components and multiplexed onto one of three respectiveoutput channels 366, 368, 370. Each of the output channels 366, 368, 370therefore presents an over sampled version comprising base and extrasignal samples for one of the three components of the legalised signalsamples. These signal samples are fed to a first input of a decimatingfilter 372. The decimating processor 178, also receives at a secondinput the modified flags which were generated within the color legaliser162. The flags are received from the connecting channel 176 by anassignment processor 374 which operates to assign each of the modifiedflags to a corresponding one of the three decimating filters, inaccordance with which of the three signal components the flags weregenerated. A copy of the modified flags as received on the connectingchannel 176 are fed to the connecting channel 188, coupled to thecorresponding output of the decimating processor 178.

One of the decimating filters 372 is shown in more detail in FIG. 24,where parts also appearing in FIG. 23, bear the same numericaldesignations. In FIG. 24, the modified flags are fed to a first shiftregister 376 which has a number of stages 377 in which the flagssuccessively fed to the shift register are stored. In a similar mannerthe signal samples for the component of the legalised color signalsamples for which the filter is provided are fed to a second shiftregister 378 having a corresponding number of stages 379. Each of thestages 379, equally displaced with respect to a centre stage 391 of theshift registers 378 are paired, except for the centre stage. The contentof the first stage of each pair is fed to a first input of an associatedadder 400, and the second stage of each pair is fed to a second input ofthe associated adder 400. An output from each of the adders 400 is fedto a first input of an associated multiplier 402. The multipliersoperate to scale the summed contents of the corresponding shift registerpair with one of a plurality of scaling coefficients d₀, d₁, d₂, d₃, d₄,d₅ of a window function. The scaled summed signal samples are thensummed by a network of adders 404, to form a combined signal samplewhich is fed to a first input of a final adder 406. To a second input ofthe final adder 406, the signal sample contained in the centre stage 391of the shift register 378 is fed, and summed with the combined signalsample to produce a composite decimated signal sample which is fed to afirst input of a logic gate 408. The signal sample of the centre stage391 of the shift register 378 is also fed to a second input of the gate408. Each stage of the second shift register is connected to a logic‘OR’ function unit 410, which generates a logic output signal which isfed to a third control input of the logic gate 408. The logic gate 408presents a decimated sample on the output conductor 412.

In operation, the signal samples are fed to the second shift register insuccession, and are combined by the adders 400, and scaled by the windowfunction formed by the coefficients d₀, d₁, d₂, d₃, d₄, d₅, and summedto formed the composite decimated output signal sample fed to the logicgate 408. The corresponding modified flags are logic ‘OR-ed’ todetermine whether any of the legalised color signal samples within theshift register 378 have been changed by the color legaliser 162 from theinput signal samples. If none of the legalised color signal samples havechanged with respect to the input signal samples then the control signalfed to the logic gate 408 is true, which sets the gate to feed thesignal sample from the centre stage 391 of the shift register 378 to theoutput conductor 412. If however any of the flags in the first shiftregister 376 have been set ‘FALSE’, indicating that a legalised colorsignal sample has changed with respect to the input signal samples, thenthe control signal fed to the logic gate is set ‘FALSE’ with the resultthat the composite signal sample is selected as the decimated signalsample. Furthermore by feeding two samples to the second shift registerfor every decimated signal sample formed, the signal is decimated from8:8:8 to 4:4:4.

The second shift register 378, the adders 400 and the associatedmultipliers 402 combine to perform a combined decimating and filteringprocess which generates at the output of the final adder 406 a decimatedsignal sample. However if none of the legalised color signal samples,within the corresponding memory length or constraint length of thesecond shift register 378 have changed with respect to the input signalsamples, then the modified flags which are ‘OR-ed’ and fed to thecontrol input of the gate 408 arrange for the signal sample held in thecentre stage to be fed to the output conductor 412. This arrangement ofbypassing the decimating filter provides a further advantage to thesecond embodiment of the invention which is associated with thepossibility of the signal samples which are combined by the decimatingfilter being made once again into illegal color pixels. By determiningwhether either of the base or the extra legalised color signal sampleswithin a window which corresponds to the constraint length of thedecimating filter were changed by the color legaliser 162 and formingthe decimated signal sample from the base legalised color signal sample,a risk of producing illegal pixels as a result of combining the base andthe legalised color signal samples into a composite signal is reduced.At the output of the decimating processor 178, the multiplexer 414 againmultiplexes the components of the pixels of the legal color video imageto form a stream of data in the format 4:4:4 for the three YUV or RGBcomponents corresponding to the color difference signals.

The legalised color signal samples are fed from the decimating processor178 via the channel 184 to the second color reference converter 182. Thesecond color reference converter 182 operates to convert the legalisedcolor signal samples from the RGB color reference space to the YUV colordifference space in the case where either of the two color legalisationprocesses for RGB signal samples were used. If one of the two othercolor legalising methods for YUV signal samples were chosen whichrequire the signal samples to remain with components corresponding tothe YUV color difference reference space, then the conversion processprovided by the color conversion processor 182 is bypassed. In eithercase the legalised color signal samples are presented as YUV colordifference signals at an output of the conversion processor 182 and fedvia the connecting channels 190, to the color anti-aliasing processor180.

The color anti-aliasing processor 180 is shown in more detail in FIG. 25where parts also appearing in FIG. 11 bear the same numericaldesignations. As shown in FIG. 25, the color anti-aliasing processor 180the legalised color signal samples in rate 4:4:4 are fed to an input ofa de-multiplexer 416. The de-multiplexer 416 operates to separate thesignal samples associated with each of the three YUV color differencesignal components. The luminance component Y is fed to an outputmultiplexer 418, whereas the red and blue color difference signalsamples are fed respectively to an input of one of two correspondinganti-aliasing filters 420. Each of the anti-aliasing filters receivesone of the two chrominance U,V components of the legalised color signalsamples at an input, and in parallel the signal samples are fed to afirst output terminal 421 of a switch 424. The anti-aliasing filters 420operate to filter the signal samples in accordance with a low passfilter characteristic having a cut-off frequency approximately at halfthe value of the sampling frequency of the chrominance signals in 4:2:2rate in order to substantially reduce aliasing errors in the chrominancesignal samples in preparation for decimation from four samples (4:4:4)to two samples (4:2:2). The filtered signal samples remain however inover sampled form (4:4:4) having four samples per pixel and are fed to asecond terminal 412 of the switch 424. The switch 424 is controlled by acontrol processor 426 from a control bus 428. The control processor 426receives via the channel 188 the data which is representative of themodified flags which were generated in the color legaliser 162 and fedvia the decimating processor 178 to the color anti aliasing filter. In asimilar operation to that performed within the decimating processor 178,the control processor 426 operates to examine the modified flags withina window corresponding to the constraint length of the anti-aliasingfilter 420, to determine whether the legalised signal samples have beenchanged by the color legaliser 162 with respect to the correspondinginput signal samples. If any of the legalised signal samples within thewindow have been changed with respect to the input signal samples, asindicated by one of the modified flags being set ‘FALSE’, then theswitch 426 is set to the second of the input terminals 422 of the switch424, and the filtered chrominance signal samples are fed from the outputof the anti-aliasing filter 420 to the output multiplexer 418. Ifhowever none of the modified flags within the window corresponding tothe constraint length of the anti-aliasing filter are ‘FALSE’ indicatingthat none of the legalised color signal samples have been changed withreference to the input signal samples then the control processor 426operates to feed appropriate control signals via a control bus 428 toset the switch 424 to the first terminal 421 so that the signal samplepresent at the second input terminal 421 is fed to the outputmultiplexer 418.

As already explained, the color anti-aliasing filter 180, operates tofilter the chrominance signal samples before the chrominance signalsamples are decimated from four samples to two samples to form the 4:2:2format. To this end, the control processor 426 operates to select eitherthe un-filtered chrominance signal sample or the filtered chrominancesignal sample, in dependence upon whether any of the corresponding inputsignal samples within the window corresponding to the constraint lengthof the filter has been changed in the legalised version of the signalsample, as indicated by the modified flags. This has a particularadvantage because it has been found that one effect of filtering thelegalised version of the signal samples can be to once againre-illegalise the color pixel represented by the signal samples or tomake an originally legal color pixel illegal. By generating the modifiedflags which are received at the control processor 426 which indicatewhether the legalised signal samples within the window have changed ornot, an efficient process for bypassing the anti-aliasing filter isprovided if the legalised signal samples have not changed with respectto input signal samples. After passing through or bypassing theanti-aliasing signal samples the YUV color difference signal samples arere-multiplexed by the output multiplexer 418 operates to re-multiplexthe signal samples from each of the signal components to form the YUVcolor difference signals in 4:4:4 form. These legalised color signalsamples are then fed to the second adjustment factor generator 196 viathe channel 194.

The second adjustment factor generator 196, the second color legaliser204 and the second adjustment factor softener 200 operate in combinationto provide a second color legalising process to the color signal samplesreceived from the decimating processor 180. Although the color signalsamples may be legal after the color legaliser 162 has processed theinput signal samples, it has been observed that the operations of theanti-aliasing filter, the second color reference converter 182, thedecimating processor 180 subsequently applied to the color signalsamples produced at the output of the first color legaliser 162 can havean effect of altering some of these signal samples. As a result, thecorresponding pixels in the RGB color reference space produced fromthese components can be once again illegal colors because theycorrespond to points outside the RGB color reference space. To remedythis problem and to provide a further improvement to the imageprocessing apparatus shown in FIG. 11, a second stage of colorlegalisation is applied to the signal samples received from thedecimating processor 180 and this is afforded by the second adjustmentfactor generator 196 in combination with the second color legaliser 204in accordance with one of the methods of legalising the color imagedescribed above. Once again however, the adjustment factors provided bythe second adjustment factor generator 196 via the connecting channel198 are softened by the second adjustment factor softener 200 whichreceives the adjustment factors at an input from the connecting channel198. The second softener is implemented in substantially the same formas the softening filter 156 but will have a smaller window length. Thecolor signal samples are fed directly to the second color legaliser 204via the second connecting channel 202. The second color legaliseroperates to filter the adjustment factors using the same legalisingmethod applied by the first color legaliser 162. The softened adjustmentfactors are then fed to a second input of the second color legaliser 204via the channel 206. The second color legaliser 204 then combines thesoftened adjustment factors with the color signal samples and providesat an output of the second color legaliser 204 final legalised colorsignal samples. These are fed to the input of the color differenceconversion processor 208 via the channel 210. If the legalising methodis applied in the RGB-signal format, then the second color referenceconverter 182, would not convert the signal samples to YUV form, but thelegalised color signal samples would remain in RGB form, and would beconverted to YUV form by the second color conversion processor 208. Thisfinal color conversion processor 208 converts the color differencesignal samples in YUV format to YCrCb format in accordance withequations (2) and (3). The color conversion processor 208 also operatesto decimate the red and blue chrominance signal samples to the effect ofhalving the sampling rate of these two chrominance signal samples sothat at the output of the processor channel 211, the legalised colorsignal samples are once again in the CCIR-601 4:2:2 format. Finally thelegalised color signal samples are duplicated by the duplicator 215which operates to feed copies of the legalised color signal samples inthe 4:2:2 format to each of the two outputs 12 and 14. A further outputchannel 19 from the image processing apparatus 1 shown in FIG. 11 isprovided from a second output from the decimating processor 180 providesa copy of the data representative of the modified flags. These modifiedflags are fed to the host control processor 16 which is shown in FIG. 3and which operates to display on the visual display unit 20 arepresentation of the corresponding location and value of those pixelsof the color image which were illegal.

As will be appreciated by those skilled in the art, variousmodifications may be made to the example embodiments without departingfrom the scope of the present invention. In particular whilst thepreferred embodiments have been described with reference to signalsamples in the form of color difference signal samples having colordifference components, it will be appreciated that the image processingapparatus can operate with color signal samples representative of acolor video image in any format. Furthermore it will be understood thatwhilst the embodiments of the invention have been described in a form inwhich an image processing apparatus operates to perform certainfunctions, it will be understood that the embodiments of the inventioncould be implemented in the form of dedicated hardware or alternativelycould be implemented as a data processor or a set of data processorsoperating to fulfil the function of the features of the embodiments byexecuting appropriate software. It will therefore be appreciated that acomputer programme providing the function of these features whenexecuted on a data processor or a set of data processors and a storagemedium on which such a computer programme may be stored are envisaged asaspects of the present invention.

We claim:
 1. A method of processing input signal samples representativeof at least part of a color video image to produce legalised signalsamples representative of a legal color version of said image, saidmethod comprising the steps of; generating an over sampled version ofthe input signal samples by generating at least one extra signal samplefor each base input signal sample, generating adjustment factors fromsaid input signal samples, which when combined with said input signalsamples have an effect of converting illegal color pixels of said colorvideo image into legal color pixels; combining said adjustment factorswith said input signal samples to produce said legalised color signalsamples; decimating said legalised color signal samples to producelegalised signal samples having a sampling frequency corresponding tothat of the base input signal samples; generating further adjustmentfactors in dependence upon said decimated legalised color signalsamples, and combining said further adjustment factors with saiddecimated legalised color signal samples.
 2. A method as claimed inclaim 1, wherein before the step of combining said further adjustmentfactors with said decimated legalised color signal samples, thereincludes the further step of softening said further adjustment factors.3. A method as claimed in claim 1, including, before the step ofdecimating said legalised signal samples, the step of filtering saidlegalised color signal samples with an anti-aliasing filter, having aband width substantially equal to half the sampling frequency used torepresent the legalised color signal samples.
 4. A method as claimed inclaim 1, wherein said input signal samples are color difference signalsamples having a luminance and two color difference components, saidmethod comprising the steps of converting the input color differencesignal samples into a color reference signal samples having componentsrepresentative of three orthogonal color reference axes of red, greenand blue; combining said color reference signal samples with saidadjustment factors; and converting said combined color reference signalsamples into color difference signal samples.
 5. A method as claimed inclaim 1, wherein said adjustment factors are scaling factors, and thestep of combining said adjustment factors with said input signal samplescomprises multiplying said adjustment factors with said color differencesignal sample.
 6. An image processing apparatus which operates toprocess signal samples representative of at least part of a color videoimage to produce legal color signal samples representative of a legalcolor version of said image, said apparatus comprising an over samplingprocessor which operates to generate an over sampled version of theinput signal samples by generating at least one extra signal sample foreach base input signal sample, an adjustment factor generator, whichoperates to generate a plurality of adjustment factors which whencombined with said input signal samples have an effect of convertingillegal color pixels of said color image into legal color pixels; acolor legaliser coupled to said adjustment factor generator, whichoperates to combine said adjustment factors with said input signalsamples to produce the legalised color signal samples, a decimatingprocessor coupled to the color legaliser which operates to decimate saidlegalised color signal samples to produce legalised signal sampleshaving a sampling rate corresponding to that of the base input signalsamples, a further adjustment factor generator and a further colorlegaliser each of which is coupled to the output of said decimatingprocessor, said further adjustment factor generator operating togenerate further adjustment factors, and said further color legaliseroperating to combine said further adjustment factors with said decimatedlegalised color signal samples.
 7. An image processing apparatus asclaimed in claim 6, comprising a further adjustment factor softenercoupled between said further adjustment factor generator and saidfurther color legaliser, said softener operating to adapt said furtheradjustment factors to reduce a possibility distortion caused bycombining said further adjustment factors with said legalised colorsignal samples, wherein the softened further adjustment factors are fedto said further color legaliser and combined with said decimatedlegalised color signal samples to produce final legalised color signalsamples.
 8. An image processing apparatus as claimed in claim 6,comprising an anti-aliasing processor coupled between said colorlegaliser and said decimating processor, which operates to filter theover sampled version of said legalised color signal samples before beingdecimated by said decimating filter.
 9. An image processing apparatus asclaimed in claim 6, wherein said input signal samples are colordifference signals having a luminance and two color differencecomponents, said apparatus comprising a color conversion processorcoupled in operative association with said adjustment factor generator,which operates to generate a version of said input signal samples in theform of color reference signal samples having components incorresponding to red, green and blue light by converting said inputsignal samples in the form of color difference signal samples, saidversion of said input signal samples in color reference form and theversion in the color difference form being feed in parallel to saidadjustment factor generator.
 10. An image processing apparatus asclaimed in claim 6 wherein said adjustment factors are scaling factorsbetween zero and one, said color legaliser operating to multiply saidscaling factors with said input signal samples.
 11. A computer programproduct comprising a computer readable carrier having stored thereon acomputer program, which when loaded on to a computer performs the stepsof the method according to claim
 1. 12. A computer programmed with acomputer program according to claim
 11. 13. A video signal processingsystem comprising a video reproducing apparatus operable to reproducevideo signal samples representative of at least part of a color videoimage, an image processing apparatus which operates to process saidvideo signal samples to produce legal color signal samplesrepresentative of a legal color version of said image, said imageprocessing apparatus comprising an over sampling processor whichoperates to generate an over sampled version of the input signal samplesby generating at least one extra signal sample for each base inputsignal sample, an adjustment factor generator, which operates togenerate a plurality of adjustment factors which when combined with saidinput signal samples have an effect of converting illegal color pixelsof said color image into legal color pixels; a color legaliser coupledto said adjustment factor generator, which operates to combine saidadjustment factors with said input signal samples to produce thelegalised color signal samples, a decimating processor coupled to thecolor legaliser which operates to decimate said legalised color signalsamples to produce legalised signal samples having a sampling ratecorresponding to that of the base input signal samples, a furtheradjustment factor generator and a further color legaliser each of whichis coupled to the output of said decimating processor, said furtheradjustment factor generator operating to generate further adjustmentfactors, and said further color legaliser operating to combine saidfurther adjustment factors with said decimated legalised color signalsamples.
 14. A video signal processing system as claimed in claim 13,comprising a display means which is arranged in operation to displaysaid legalised color signal samples.
 15. A video signal processingsystem as claimed in claim 13, wherein said reproducing apparatus isalso a recording apparatus and said legalised color signal samples arerecorded onto a readable medium by said recording/reproducing apparatus.16. A video signal processing system as claimed in claim 13, comprisinga recording apparatus, wherein said legalised color signal samples arerecorded onto a recordable medium by said recording apparatus.